{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonsai in Tensorflow\n",
    "\n",
    "This is a simple notebook that illustrates the usage of Tensorflow implementation of Bonsai. We are using the USPS dataset. Please refer to `fetch_usps.py` and run it for downloading and cleaning up the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-15T12:06:06.056404Z",
     "start_time": "2018-08-15T12:06:05.112969Z"
    }
   },
   "outputs": [],
   "source": [
    "# Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "# Licensed under the MIT license.\n",
    "\n",
    "import helpermethods\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import sys\n",
    "import os\n",
    "\n",
    "#Provide the GPU number to be used\n",
    "os.environ['CUDA_VISIBLE_DEVICES'] =''\n",
    "\n",
    "#Bonsai imports\n",
    "from edgeml_tf.trainer.bonsaiTrainer import BonsaiTrainer\n",
    "from edgeml_tf.graph.bonsai import Bonsai\n",
    "\n",
    "# Fixing seeds for reproducibility\n",
    "tf.set_random_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# USPS Data\n",
    "\n",
    "It is assumed that the USPS data has already been downloaded and set up with the help of [fetch_usps.py](fetch_usps.py) and is present in the `./usps10` subdirectory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-15T12:06:06.104645Z",
     "start_time": "2018-08-15T12:06:06.058368Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Dimension:  257\n",
      "Num classes:  10\n"
     ]
    }
   ],
   "source": [
    "#Loading and Pre-processing dataset for Bonsai\n",
    "dataDir = \"usps10/\"\n",
    "(dataDimension, numClasses, Xtrain, Ytrain, Xtest, Ytest, mean, std) = helpermethods.preProcessData(dataDir, isRegression=False)\n",
    "print(\"Feature Dimension: \", dataDimension)\n",
    "print(\"Num classes: \", numClasses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Parameters\n",
    "\n",
    "Note that Bonsai is designed for low-memory setting and the best results are obtained when operating in that setting. Use the sparsity, projection dimension and tree depth to vary the model size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-15T12:06:06.123318Z",
     "start_time": "2018-08-15T12:06:06.106847Z"
    }
   },
   "outputs": [],
   "source": [
    "sigma = 1.0 #Sigmoid parameter for tanh\n",
    "depth = 3 #Depth of Bonsai Tree\n",
    "projectionDimension = 28 #Lower Dimensional space for Bonsai to work on\n",
    "\n",
    "#Regularizers for Bonsai Parameters\n",
    "regZ = 0.0001\n",
    "regW = 0.001\n",
    "regV = 0.001\n",
    "regT = 0.001\n",
    "\n",
    "totalEpochs = 100\n",
    "\n",
    "learningRate = 0.01\n",
    "\n",
    "outFile = None\n",
    "\n",
    "#Sparsity for Bonsai Parameters. x => 100*x % are non-zeros\n",
    "sparZ = 0.2\n",
    "sparW = 0.3\n",
    "sparV = 0.3\n",
    "sparT = 0.62\n",
    "\n",
    "batchSize = np.maximum(100, int(np.ceil(np.sqrt(Ytrain.shape[0]))))\n",
    "\n",
    "useMCHLoss = True #only for Multiclass cases True: Multiclass-Hing Loss, False: Cross Entropy. \n",
    "\n",
    "#Bonsai uses one classier for Binary, thus this condition\n",
    "if numClasses == 2:\n",
    "    numClasses = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Placeholders for Data feeding during training and infernece"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-15T12:06:06.220274Z",
     "start_time": "2018-08-15T12:06:06.125219Z"
    }
   },
   "outputs": [],
   "source": [
    "X = tf.placeholder(\"float32\", [None, dataDimension])\n",
    "Y = tf.placeholder(\"float32\", [None, numClasses])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a directory for current model in the datadirectory using timestamp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-15T12:06:06.264985Z",
     "start_time": "2018-08-15T12:06:06.222170Z"
    }
   },
   "outputs": [],
   "source": [
    "currDir = helpermethods.createTimeStampDir(dataDir)\n",
    "helpermethods.dumpCommand(sys.argv, currDir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonsai Graph Object\n",
    "\n",
    "Instantiating the Bonsai Graph which will be used for training and inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-15T12:06:06.341168Z",
     "start_time": "2018-08-15T12:06:06.266877Z"
    }
   },
   "outputs": [],
   "source": [
    "bonsaiObj = Bonsai(numClasses, dataDimension, projectionDimension, depth, sigma)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonsai Trainer Object\n",
    "\n",
    "Instantiating the Bonsai Trainer which will be used for 3 phase training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-15T12:06:07.973584Z",
     "start_time": "2018-08-15T12:06:06.342945Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\t-vekusu\\AppData\\Local\\Continuum\\anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\ops\\gradients_impl.py:100: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    }
   ],
   "source": [
    "bonsaiTrainer = BonsaiTrainer(bonsaiObj, regW, regT, regV, regZ, sparW, sparT, sparV, sparZ,\n",
    "                              learningRate, X, Y, useMCHLoss, outFile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Session declaration and variable initialization. \n",
    "Interactive Session doesn't clog the entire GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-15T12:06:15.577425Z",
     "start_time": "2018-08-15T12:06:07.976090Z"
    }
   },
   "outputs": [],
   "source": [
    "sess = tf.InteractiveSession()\n",
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonsai Training Routine\n",
    "\n",
    "The method to to run the 3 phase training, followed by giving out the best early stopping model, accuracy along with saving of the parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-15T12:07:02.500241Z",
     "start_time": "2018-08-15T12:06:15.579618Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch Number: 0\n",
      "\n",
      "******************** Dense Training Phase Started ********************\n",
      "\n",
      "\n",
      "Classification Train Loss: 6.388934433460236\n",
      "Training accuracy (Classification): 0.6250000005174015\n",
      "Test accuracy 0.726956\n",
      "MarginLoss + RegLoss: 1.4466879 + 3.6487768 = 5.0954647\n",
      "\n",
      "\n",
      "Epoch Number: 1\n",
      "\n",
      "Classification Train Loss: 3.6885906954606376\n",
      "Training accuracy (Classification): 0.8623611107468605\n",
      "Test accuracy 0.758346\n",
      "MarginLoss + RegLoss: 1.0173264 + 2.778634 = 3.7959604\n",
      "\n",
      "\n",
      "Epoch Number: 2\n",
      "\n",
      "Classification Train Loss: 2.667721450328827\n",
      "Training accuracy (Classification): 0.9184722271230485\n",
      "Test accuracy 0.7429\n",
      "MarginLoss + RegLoss: 0.92546654 + 2.095467 = 3.0209336\n",
      "\n",
      "\n",
      "Epoch Number: 3\n",
      "\n",
      "Classification Train Loss: 1.9921080254846149\n",
      "Training accuracy (Classification): 0.941944446000788\n",
      "Test accuracy 0.767314\n",
      "MarginLoss + RegLoss: 0.7603649 + 1.5889603 = 2.3493252\n",
      "\n",
      "\n",
      "Epoch Number: 4\n",
      "\n",
      "Classification Train Loss: 1.5233625107341342\n",
      "Training accuracy (Classification): 0.9563888907432556\n",
      "Test accuracy 0.791231\n",
      "MarginLoss + RegLoss: 0.6496898 + 1.2271981 = 1.8768879\n",
      "\n",
      "\n",
      "Epoch Number: 5\n",
      "\n",
      "Classification Train Loss: 1.1950715631246567\n",
      "Training accuracy (Classification): 0.9650000035762787\n",
      "Test accuracy 0.810164\n",
      "MarginLoss + RegLoss: 0.54003507 + 0.97295314 = 1.5129882\n",
      "\n",
      "\n",
      "Epoch Number: 6\n",
      "\n",
      "Classification Train Loss: 0.9672323316335678\n",
      "Training accuracy (Classification): 0.968333340353436\n",
      "Test accuracy 0.855007\n",
      "MarginLoss + RegLoss: 0.44149697 + 0.79325426 = 1.2347512\n",
      "\n",
      "\n",
      "Epoch Number: 7\n",
      "\n",
      "Classification Train Loss: 0.8014380658666292\n",
      "Training accuracy (Classification): 0.9722222313284874\n",
      "Test accuracy 0.874938\n",
      "MarginLoss + RegLoss: 0.37062877 + 0.6628879 = 1.0335166\n",
      "\n",
      "\n",
      "Epoch Number: 8\n",
      "\n",
      "Classification Train Loss: 0.684503066043059\n",
      "Training accuracy (Classification): 0.976111119820012\n",
      "Test accuracy 0.899851\n",
      "MarginLoss + RegLoss: 0.3099702 + 0.5688073 = 0.8787775\n",
      "\n",
      "\n",
      "Epoch Number: 9\n",
      "\n",
      "Classification Train Loss: 0.5987317487597466\n",
      "Training accuracy (Classification): 0.9794444565971693\n",
      "Test accuracy 0.907324\n",
      "MarginLoss + RegLoss: 0.2689218 + 0.49965328 = 0.7685751\n",
      "\n",
      "\n",
      "Epoch Number: 10\n",
      "\n",
      "Classification Train Loss: 0.5343128165437115\n",
      "Training accuracy (Classification): 0.9804166778922081\n",
      "Test accuracy 0.9143\n",
      "MarginLoss + RegLoss: 0.24538836 + 0.44663915 = 0.6920275\n",
      "\n",
      "\n",
      "Epoch Number: 11\n",
      "\n",
      "Classification Train Loss: 0.48874612069792217\n",
      "Training accuracy (Classification): 0.9801388987236552\n",
      "Test accuracy 0.916293\n",
      "MarginLoss + RegLoss: 0.23703864 + 0.40629783 = 0.6433365\n",
      "\n",
      "\n",
      "Epoch Number: 12\n",
      "\n",
      "Classification Train Loss: 0.44733552055226433\n",
      "Training accuracy (Classification): 0.98097223126226\n",
      "Test accuracy 0.918286\n",
      "MarginLoss + RegLoss: 0.23851919 + 0.37269312 = 0.6112123\n",
      "\n",
      "\n",
      "Epoch Number: 13\n",
      "\n",
      "Classification Train Loss: 0.4165669356783231\n",
      "Training accuracy (Classification): 0.9822222317258517\n",
      "Test accuracy 0.917289\n",
      "MarginLoss + RegLoss: 0.23061273 + 0.345445 = 0.57605773\n",
      "\n",
      "\n",
      "Epoch Number: 14\n",
      "\n",
      "Classification Train Loss: 0.39181090601616436\n",
      "Training accuracy (Classification): 0.9812500087751282\n",
      "Test accuracy 0.92277\n",
      "MarginLoss + RegLoss: 0.2121576 + 0.32245666 = 0.53461426\n",
      "\n",
      "\n",
      "Epoch Number: 15\n",
      "\n",
      "Classification Train Loss: 0.36949437111616135\n",
      "Training accuracy (Classification): 0.9820833446251022\n",
      "Test accuracy 0.926258\n",
      "MarginLoss + RegLoss: 0.19854721 + 0.30341443 = 0.50196165\n",
      "\n",
      "\n",
      "Epoch Number: 16\n",
      "\n",
      "Classification Train Loss: 0.3469446731938256\n",
      "Training accuracy (Classification): 0.9831944538487328\n",
      "Test accuracy 0.927255\n",
      "MarginLoss + RegLoss: 0.19628116 + 0.28535655 = 0.48163772\n",
      "\n",
      "\n",
      "Epoch Number: 17\n",
      "\n",
      "Classification Train Loss: 0.329777576857143\n",
      "Training accuracy (Classification): 0.984166675971614\n",
      "Test accuracy 0.92277\n",
      "MarginLoss + RegLoss: 0.20166817 + 0.26965213 = 0.4713203\n",
      "\n",
      "\n",
      "Epoch Number: 18\n",
      "\n",
      "Classification Train Loss: 0.317672994815641\n",
      "Training accuracy (Classification): 0.9815277879436811\n",
      "Test accuracy 0.925262\n",
      "MarginLoss + RegLoss: 0.20086277 + 0.2559616 = 0.45682436\n",
      "\n",
      "\n",
      "Epoch Number: 19\n",
      "\n",
      "Classification Train Loss: 0.3000084459781647\n",
      "Training accuracy (Classification): 0.9843055655558904\n",
      "Test accuracy 0.931739\n",
      "MarginLoss + RegLoss: 0.18073215 + 0.24324338 = 0.42397553\n",
      "\n",
      "\n",
      "Epoch Number: 20\n",
      "\n",
      "Classification Train Loss: 0.2897499371320009\n",
      "Training accuracy (Classification): 0.9827777867515882\n",
      "Test accuracy 0.921276\n",
      "MarginLoss + RegLoss: 0.20172484 + 0.23221089 = 0.43393573\n",
      "\n",
      "\n",
      "Epoch Number: 21\n",
      "\n",
      "Classification Train Loss: 0.2821065636558665\n",
      "Training accuracy (Classification): 0.9812500096029706\n",
      "Test accuracy 0.928749\n",
      "MarginLoss + RegLoss: 0.18990344 + 0.22147894 = 0.41138238\n",
      "\n",
      "\n",
      "Epoch Number: 22\n",
      "\n",
      "Classification Train Loss: 0.2660716378854381\n",
      "Training accuracy (Classification): 0.9844444559680091\n",
      "Test accuracy 0.928251\n",
      "MarginLoss + RegLoss: 0.17955597 + 0.21111046 = 0.39066643\n",
      "\n",
      "\n",
      "Epoch Number: 23\n",
      "\n",
      "Classification Train Loss: 0.2567368100086848\n",
      "Training accuracy (Classification): 0.9852777885066138\n",
      "Test accuracy 0.928251\n",
      "MarginLoss + RegLoss: 0.18770447 + 0.20248988 = 0.39019436\n",
      "\n",
      "\n",
      "Epoch Number: 24\n",
      "\n",
      "Classification Train Loss: 0.25224825532899964\n",
      "Training accuracy (Classification): 0.9823611204822859\n",
      "Test accuracy 0.932735\n",
      "MarginLoss + RegLoss: 0.18552671 + 0.19460817 = 0.38013488\n",
      "\n",
      "\n",
      "Epoch Number: 25\n",
      "\n",
      "Classification Train Loss: 0.24661735258996487\n",
      "Training accuracy (Classification): 0.9804166762365235\n",
      "Test accuracy 0.931241\n",
      "MarginLoss + RegLoss: 0.18796808 + 0.18610859 = 0.37407666\n",
      "\n",
      "\n",
      "Epoch Number: 26\n",
      "\n",
      "Classification Train Loss: 0.23342499737110403\n",
      "Training accuracy (Classification): 0.9829166763358645\n",
      "Test accuracy 0.932735\n",
      "MarginLoss + RegLoss: 0.17906994 + 0.17793566 = 0.3570056\n",
      "\n",
      "\n",
      "Epoch Number: 27\n",
      "\n",
      "Classification Train Loss: 0.22210048822065195\n",
      "Training accuracy (Classification): 0.9851388972666528\n",
      "Test accuracy 0.934728\n",
      "MarginLoss + RegLoss: 0.17679122 + 0.16876754 = 0.34555876\n",
      "\n",
      "\n",
      "Epoch Number: 28\n",
      "\n",
      "Classification Train Loss: 0.2189549288402001\n",
      "Training accuracy (Classification): 0.9831944538487328\n",
      "Test accuracy 0.932237\n",
      "MarginLoss + RegLoss: 0.19115414 + 0.16296963 = 0.35412377\n",
      "\n",
      "\n",
      "Epoch Number: 29\n",
      "\n",
      "Classification Train Loss: 0.21842483865718046\n",
      "Training accuracy (Classification): 0.9805555658208\n",
      "Test accuracy 0.936722\n",
      "MarginLoss + RegLoss: 0.17462157 + 0.15921564 = 0.3338372\n",
      "\n",
      "\n",
      "Epoch Number: 30\n",
      "\n",
      "Classification Train Loss: 0.21449942576388517\n",
      "Training accuracy (Classification): 0.9804166754086813\n",
      "Test accuracy 0.939711\n",
      "MarginLoss + RegLoss: 0.17741902 + 0.15273981 = 0.33015883\n",
      "\n",
      "\n",
      "Epoch Number: 31\n",
      "\n",
      "Classification Train Loss: 0.20739994280868107\n",
      "Training accuracy (Classification): 0.9825000100665622\n",
      "Test accuracy 0.933732\n",
      "MarginLoss + RegLoss: 0.17381513 + 0.1498537 = 0.32366884\n",
      "\n",
      "\n",
      "Epoch Number: 32\n",
      "\n",
      "Classification Train Loss: 0.20110303929282558\n",
      "Training accuracy (Classification): 0.9840277888708644\n",
      "Test accuracy 0.93423\n",
      "MarginLoss + RegLoss: 0.18619148 + 0.14583017 = 0.33202165\n",
      "\n",
      "\n",
      "Epoch Number: 33\n",
      "\n",
      "******************** IHT Phase Started ********************\n",
      "\n",
      "\n",
      "Classification Train Loss: 0.21433907147083017\n",
      "Training accuracy (Classification): 0.9801388987236552\n",
      "Test accuracy 0.927255\n",
      "MarginLoss + RegLoss: 0.19979775 + 0.12088289 = 0.32068065\n",
      "\n",
      "\n",
      "Epoch Number: 34\n",
      "\n",
      "Classification Train Loss: 0.1990115779141585\n",
      "Training accuracy (Classification): 0.980694454577234\n",
      "Test accuracy 0.933234\n",
      "MarginLoss + RegLoss: 0.17835513 + 0.12438774 = 0.30274287\n",
      "\n",
      "\n",
      "Epoch Number: 35\n",
      "\n",
      "Classification Train Loss: 0.20429682172834873\n",
      "Training accuracy (Classification): 0.9788888974322213\n",
      "Test accuracy 0.929248\n",
      "MarginLoss + RegLoss: 0.19013074 + 0.12853864 = 0.31866938\n",
      "\n",
      "\n",
      "Epoch Number: 36\n",
      "\n",
      "Classification Train Loss: 0.19357945707937083\n",
      "Training accuracy (Classification): 0.9816666767001152\n",
      "Test accuracy 0.932735\n",
      "MarginLoss + RegLoss: 0.18534705 + 0.12509713 = 0.31044418\n",
      "\n",
      "\n",
      "Epoch Number: 37\n",
      "\n",
      "Classification Train Loss: 0.18653404754069117\n",
      "Training accuracy (Classification): 0.9818055638008647\n",
      "Test accuracy 0.929746\n",
      "MarginLoss + RegLoss: 0.18708317 + 0.12236847 = 0.30945164\n",
      "\n",
      "\n",
      "Epoch Number: 38\n",
      "\n",
      "Classification Train Loss: 0.18141362298693922\n",
      "Training accuracy (Classification): 0.9815277871158388\n",
      "Test accuracy 0.933234\n",
      "MarginLoss + RegLoss: 0.18262453 + 0.11991154 = 0.30253607\n",
      "\n",
      "\n",
      "Epoch Number: 39\n",
      "\n",
      "Classification Train Loss: 0.17729416727605793\n",
      "Training accuracy (Classification): 0.9820833429694176\n",
      "Test accuracy 0.932735\n",
      "MarginLoss + RegLoss: 0.1798804 + 0.11748926 = 0.29736966\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch Number: 40\n",
      "\n",
      "Classification Train Loss: 0.17237282171845436\n",
      "Training accuracy (Classification): 0.9837500088744693\n",
      "Test accuracy 0.937718\n",
      "MarginLoss + RegLoss: 0.17473482 + 0.11479883 = 0.28953364\n",
      "\n",
      "\n",
      "Epoch Number: 41\n",
      "\n",
      "Classification Train Loss: 0.16901198805620274\n",
      "Training accuracy (Classification): 0.9837500097023116\n",
      "Test accuracy 0.93423\n",
      "MarginLoss + RegLoss: 0.17860568 + 0.112817116 = 0.29142278\n",
      "\n",
      "\n",
      "Epoch Number: 42\n",
      "\n",
      "Classification Train Loss: 0.16710670509686074\n",
      "Training accuracy (Classification): 0.9833333442608515\n",
      "Test accuracy 0.936722\n",
      "MarginLoss + RegLoss: 0.17501548 + 0.11118551 = 0.286201\n",
      "\n",
      "\n",
      "Epoch Number: 43\n",
      "\n",
      "Classification Train Loss: 0.16463725310232905\n",
      "Training accuracy (Classification): 0.9836111209458775\n",
      "Test accuracy 0.93423\n",
      "MarginLoss + RegLoss: 0.17687047 + 0.10897398 = 0.28584445\n",
      "\n",
      "\n",
      "Epoch Number: 44\n",
      "\n",
      "Classification Train Loss: 0.16215091271118987\n",
      "Training accuracy (Classification): 0.9843055663837327\n",
      "Test accuracy 0.935227\n",
      "MarginLoss + RegLoss: 0.17832607 + 0.107886344 = 0.2862124\n",
      "\n",
      "\n",
      "Epoch Number: 45\n",
      "\n",
      "Classification Train Loss: 0.16012930932144323\n",
      "Training accuracy (Classification): 0.9841666767994562\n",
      "Test accuracy 0.937718\n",
      "MarginLoss + RegLoss: 0.17309293 + 0.10644325 = 0.2795362\n",
      "\n",
      "\n",
      "Epoch Number: 46\n",
      "\n",
      "Classification Train Loss: 0.1574974125251174\n",
      "Training accuracy (Classification): 0.9850000101659033\n",
      "Test accuracy 0.93722\n",
      "MarginLoss + RegLoss: 0.17099261 + 0.10526536 = 0.27625796\n",
      "\n",
      "\n",
      "Epoch Number: 47\n",
      "\n",
      "Classification Train Loss: 0.15617641361637247\n",
      "Training accuracy (Classification): 0.9856944539480739\n",
      "Test accuracy 0.937718\n",
      "MarginLoss + RegLoss: 0.16866577 + 0.104043506 = 0.27270928\n",
      "\n",
      "\n",
      "Epoch Number: 48\n",
      "\n",
      "Classification Train Loss: 0.15530151346077523\n",
      "Training accuracy (Classification): 0.9838889001144303\n",
      "Test accuracy 0.940209\n",
      "MarginLoss + RegLoss: 0.16514857 + 0.10232182 = 0.2674704\n",
      "\n",
      "\n",
      "Epoch Number: 49\n",
      "\n",
      "Classification Train Loss: 0.15294318615148464\n",
      "Training accuracy (Classification): 0.9862500089738104\n",
      "Test accuracy 0.939711\n",
      "MarginLoss + RegLoss: 0.16788226 + 0.10096101 = 0.26884326\n",
      "\n",
      "\n",
      "Epoch Number: 50\n",
      "\n",
      "Classification Train Loss: 0.15095406781054205\n",
      "Training accuracy (Classification): 0.9861111202173762\n",
      "Test accuracy 0.940209\n",
      "MarginLoss + RegLoss: 0.17100953 + 0.10046519 = 0.27147472\n",
      "\n",
      "\n",
      "Epoch Number: 51\n",
      "\n",
      "Classification Train Loss: 0.1513558304351237\n",
      "Training accuracy (Classification): 0.9844444543123245\n",
      "Test accuracy 0.941704\n",
      "MarginLoss + RegLoss: 0.1662268 + 0.100100346 = 0.26632714\n",
      "\n",
      "\n",
      "Epoch Number: 52\n",
      "\n",
      "Classification Train Loss: 0.14914156941490042\n",
      "Training accuracy (Classification): 0.9852777876787715\n",
      "Test accuracy 0.941206\n",
      "MarginLoss + RegLoss: 0.16318396 + 0.099286705 = 0.26247066\n",
      "\n",
      "\n",
      "Epoch Number: 53\n",
      "\n",
      "Classification Train Loss: 0.1497938595712185\n",
      "Training accuracy (Classification): 0.9851388997501798\n",
      "Test accuracy 0.932735\n",
      "MarginLoss + RegLoss: 0.17166732 + 0.09957267 = 0.27124\n",
      "\n",
      "\n",
      "Epoch Number: 54\n",
      "\n",
      "Classification Train Loss: 0.15218847369154295\n",
      "Training accuracy (Classification): 0.985277786023087\n",
      "Test accuracy 0.938715\n",
      "MarginLoss + RegLoss: 0.17181182 + 0.09915227 = 0.2709641\n",
      "\n",
      "\n",
      "Epoch Number: 55\n",
      "\n",
      "Classification Train Loss: 0.14960632245573732\n",
      "Training accuracy (Classification): 0.9855555668473244\n",
      "Test accuracy 0.943697\n",
      "MarginLoss + RegLoss: 0.16333821 + 0.09872535 = 0.26206356\n",
      "\n",
      "\n",
      "Epoch Number: 56\n",
      "\n",
      "Classification Train Loss: 0.15064662312053972\n",
      "Training accuracy (Classification): 0.9852777885066138\n",
      "Test accuracy 0.942202\n",
      "MarginLoss + RegLoss: 0.16303498 + 0.09878391 = 0.2618189\n",
      "\n",
      "\n",
      "Epoch Number: 57\n",
      "\n",
      "Classification Train Loss: 0.15265570394694805\n",
      "Training accuracy (Classification): 0.9831944555044174\n",
      "Test accuracy 0.940708\n",
      "MarginLoss + RegLoss: 0.16671813 + 0.09886683 = 0.26558495\n",
      "\n",
      "\n",
      "Epoch Number: 58\n",
      "\n",
      "Classification Train Loss: 0.15230748295370075\n",
      "Training accuracy (Classification): 0.984166675971614\n",
      "Test accuracy 0.938715\n",
      "MarginLoss + RegLoss: 0.16594657 + 0.097650595 = 0.26359716\n",
      "\n",
      "\n",
      "Epoch Number: 59\n",
      "\n",
      "Classification Train Loss: 0.1514456778143843\n",
      "Training accuracy (Classification): 0.9843055647280481\n",
      "Test accuracy 0.938216\n",
      "MarginLoss + RegLoss: 0.16204405 + 0.09645542 = 0.25849947\n",
      "\n",
      "\n",
      "Epoch Number: 60\n",
      "\n",
      "Classification Train Loss: 0.15362831794967255\n",
      "Training accuracy (Classification): 0.9829166771637069\n",
      "Test accuracy 0.933732\n",
      "MarginLoss + RegLoss: 0.17626402 + 0.09787459 = 0.2741386\n",
      "\n",
      "\n",
      "Epoch Number: 61\n",
      "\n",
      "Classification Train Loss: 0.15526858448154396\n",
      "Training accuracy (Classification): 0.9813889024986161\n",
      "Test accuracy 0.933732\n",
      "MarginLoss + RegLoss: 0.17297557 + 0.09806729 = 0.27104285\n",
      "\n",
      "\n",
      "Epoch Number: 62\n",
      "\n",
      "Classification Train Loss: 0.1579084157322844\n",
      "Training accuracy (Classification): 0.9816666767001152\n",
      "Test accuracy 0.936223\n",
      "MarginLoss + RegLoss: 0.17195764 + 0.098572396 = 0.27053005\n",
      "\n",
      "\n",
      "Epoch Number: 63\n",
      "\n",
      "Classification Train Loss: 0.1566090847675999\n",
      "Training accuracy (Classification): 0.9826389013065232\n",
      "Test accuracy 0.93423\n",
      "MarginLoss + RegLoss: 0.17155647 + 0.10033124 = 0.27188772\n",
      "\n",
      "\n",
      "Epoch Number: 64\n",
      "\n",
      "Classification Train Loss: 0.1548497351921267\n",
      "Training accuracy (Classification): 0.9837500105301539\n",
      "Test accuracy 0.941704\n",
      "MarginLoss + RegLoss: 0.16137016 + 0.099378176 = 0.26074833\n",
      "\n",
      "\n",
      "Epoch Number: 65\n",
      "\n",
      "Classification Train Loss: 0.15319975931197405\n",
      "Training accuracy (Classification): 0.9829166746801801\n",
      "Test accuracy 0.939213\n",
      "MarginLoss + RegLoss: 0.16549328 + 0.09872568 = 0.26421896\n",
      "\n",
      "\n",
      "Epoch Number: 66\n",
      "\n",
      "Classification Train Loss: 0.1565150058724814\n",
      "Training accuracy (Classification): 0.9819444542129835\n",
      "Test accuracy 0.935725\n",
      "MarginLoss + RegLoss: 0.17288828 + 0.09988601 = 0.27277428\n",
      "\n",
      "\n",
      "Epoch Number: 67\n",
      "\n",
      "******************** Sparse Retraining Phase Started ********************\n",
      "\n",
      "\n",
      "Classification Train Loss: 0.15831943404757315\n",
      "Training accuracy (Classification): 0.9829166779915491\n",
      "Test accuracy 0.935725\n",
      "MarginLoss + RegLoss: 0.17936754 + 0.101812266 = 0.28117982\n",
      "\n",
      "\n",
      "Epoch Number: 68\n",
      "\n",
      "Classification Train Loss: 0.15614786164628136\n",
      "Training accuracy (Classification): 0.9838889009422727\n",
      "Test accuracy 0.931739\n",
      "MarginLoss + RegLoss: 0.17960551 + 0.101831324 = 0.28143683\n",
      "\n",
      "\n",
      "Epoch Number: 69\n",
      "\n",
      "Classification Train Loss: 0.1662438316270709\n",
      "Training accuracy (Classification): 0.9827777884072728\n",
      "Test accuracy 0.931739\n",
      "MarginLoss + RegLoss: 0.19018382 + 0.10729199 = 0.2974758\n",
      "\n",
      "\n",
      "Epoch Number: 70\n",
      "\n",
      "Classification Train Loss: 0.16005917576452097\n",
      "Training accuracy (Classification): 0.9844444518287977\n",
      "Test accuracy 0.929248\n",
      "MarginLoss + RegLoss: 0.19133526 + 0.10547125 = 0.2968065\n",
      "\n",
      "\n",
      "Epoch Number: 71\n",
      "\n",
      "Classification Train Loss: 0.15785305326183638\n",
      "Training accuracy (Classification): 0.985000009338061\n",
      "Test accuracy 0.933732\n",
      "MarginLoss + RegLoss: 0.18749763 + 0.10477199 = 0.29226962\n",
      "\n",
      "\n",
      "Epoch Number: 72\n",
      "\n",
      "Classification Train Loss: 0.15456503671076563\n",
      "Training accuracy (Classification): 0.9843055663837327\n",
      "Test accuracy 0.935227\n",
      "MarginLoss + RegLoss: 0.1811654 + 0.10317116 = 0.28433657\n",
      "\n",
      "\n",
      "Epoch Number: 73\n",
      "\n",
      "Classification Train Loss: 0.15287091862410307\n",
      "Training accuracy (Classification): 0.9848611205816269\n",
      "Test accuracy 0.934728\n",
      "MarginLoss + RegLoss: 0.17708676 + 0.101716325 = 0.27880308\n",
      "\n",
      "\n",
      "Epoch Number: 74\n",
      "\n",
      "Classification Train Loss: 0.15090375486761332\n",
      "Training accuracy (Classification): 0.9855555643637975\n",
      "Test accuracy 0.934728\n",
      "MarginLoss + RegLoss: 0.17898533 + 0.10174509 = 0.28073043\n",
      "\n",
      "\n",
      "Epoch Number: 75\n",
      "\n",
      "Classification Train Loss: 0.15054931139780414\n",
      "Training accuracy (Classification): 0.9848611197537847\n",
      "Test accuracy 0.93722\n",
      "MarginLoss + RegLoss: 0.17272809 + 0.101017065 = 0.27374515\n",
      "\n",
      "\n",
      "Epoch Number: 76\n",
      "\n",
      "Classification Train Loss: 0.14770951929191747\n",
      "Training accuracy (Classification): 0.9855555651916398\n",
      "Test accuracy 0.936722\n",
      "MarginLoss + RegLoss: 0.17685911 + 0.09888628 = 0.2757454\n",
      "\n",
      "\n",
      "Epoch Number: 77\n",
      "\n",
      "Classification Train Loss: 0.14727520239022043\n",
      "Training accuracy (Classification): 0.9841666767994562\n",
      "Test accuracy 0.935725\n",
      "MarginLoss + RegLoss: 0.1720485 + 0.09774725 = 0.26979575\n",
      "\n",
      "\n",
      "Epoch Number: 78\n",
      "\n",
      "Classification Train Loss: 0.1471475510754519\n",
      "Training accuracy (Classification): 0.9858333418766657\n",
      "Test accuracy 0.940209\n",
      "MarginLoss + RegLoss: 0.16558117 + 0.09803399 = 0.26361516\n",
      "\n",
      "\n",
      "Epoch Number: 79\n",
      "\n",
      "Classification Train Loss: 0.14565238232413927\n",
      "Training accuracy (Classification): 0.9861111210452186\n",
      "Test accuracy 0.937718\n",
      "MarginLoss + RegLoss: 0.17031503 + 0.09688788 = 0.2672029\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch Number: 80\n",
      "\n",
      "Classification Train Loss: 0.14349345521380505\n",
      "Training accuracy (Classification): 0.9861111185616918\n",
      "Test accuracy 0.941206\n",
      "MarginLoss + RegLoss: 0.16280341 + 0.09526416 = 0.25806758\n",
      "\n",
      "\n",
      "Epoch Number: 81\n",
      "\n",
      "Classification Train Loss: 0.14298133655554718\n",
      "Training accuracy (Classification): 0.9848611205816269\n",
      "Test accuracy 0.935725\n",
      "MarginLoss + RegLoss: 0.16992427 + 0.095204785 = 0.26512906\n",
      "\n",
      "\n",
      "Epoch Number: 82\n",
      "\n",
      "Classification Train Loss: 0.1410345918395453\n",
      "Training accuracy (Classification): 0.9854166756073633\n",
      "Test accuracy 0.937718\n",
      "MarginLoss + RegLoss: 0.16711517 + 0.09361006 = 0.26072523\n",
      "\n",
      "\n",
      "Epoch Number: 83\n",
      "\n",
      "Classification Train Loss: 0.14173460192978382\n",
      "Training accuracy (Classification): 0.9858333418766657\n",
      "Test accuracy 0.935227\n",
      "MarginLoss + RegLoss: 0.17255607 + 0.09335034 = 0.26590642\n",
      "\n",
      "\n",
      "Epoch Number: 84\n",
      "\n",
      "Classification Train Loss: 0.1413275660533044\n",
      "Training accuracy (Classification): 0.985000009338061\n",
      "Test accuracy 0.939213\n",
      "MarginLoss + RegLoss: 0.1691187 + 0.09220875 = 0.26132745\n",
      "\n",
      "\n",
      "Epoch Number: 85\n",
      "\n",
      "Classification Train Loss: 0.1399904629215598\n",
      "Training accuracy (Classification): 0.9863888977302445\n",
      "Test accuracy 0.937718\n",
      "MarginLoss + RegLoss: 0.16878359 + 0.09304918 = 0.26183277\n",
      "\n",
      "\n",
      "Epoch Number: 86\n",
      "\n",
      "Classification Train Loss: 0.14306676108390093\n",
      "Training accuracy (Classification): 0.9848611214094691\n",
      "Test accuracy 0.933732\n",
      "MarginLoss + RegLoss: 0.17234829 + 0.09307802 = 0.2654263\n",
      "\n",
      "\n",
      "Epoch Number: 87\n",
      "\n",
      "Classification Train Loss: 0.14483444765210152\n",
      "Training accuracy (Classification): 0.9838888976309035\n",
      "Test accuracy 0.932237\n",
      "MarginLoss + RegLoss: 0.17103034 + 0.093002975 = 0.26403332\n",
      "\n",
      "\n",
      "Epoch Number: 88\n",
      "\n",
      "Classification Train Loss: 0.1426364007509417\n",
      "Training accuracy (Classification): 0.9854166772630479\n",
      "Test accuracy 0.938216\n",
      "MarginLoss + RegLoss: 0.17191838 + 0.09332408 = 0.26524246\n",
      "\n",
      "\n",
      "Epoch Number: 89\n",
      "\n",
      "Classification Train Loss: 0.1419605797984534\n",
      "Training accuracy (Classification): 0.9854166756073633\n",
      "Test accuracy 0.93722\n",
      "MarginLoss + RegLoss: 0.16863512 + 0.09229554 = 0.26093066\n",
      "\n",
      "\n",
      "Epoch Number: 90\n",
      "\n",
      "Classification Train Loss: 0.1416015759524372\n",
      "Training accuracy (Classification): 0.984166675971614\n",
      "Test accuracy 0.935227\n",
      "MarginLoss + RegLoss: 0.17089692 + 0.0915688 = 0.26246572\n",
      "\n",
      "\n",
      "Epoch Number: 91\n",
      "\n",
      "Classification Train Loss: 0.1449494053506189\n",
      "Training accuracy (Classification): 0.9843055663837327\n",
      "Test accuracy 0.933234\n",
      "MarginLoss + RegLoss: 0.17210826 + 0.092280895 = 0.26438916\n",
      "\n",
      "\n",
      "Epoch Number: 92\n",
      "\n",
      "Classification Train Loss: 0.14661915486471522\n",
      "Training accuracy (Classification): 0.9826388971673118\n",
      "Test accuracy 0.935725\n",
      "MarginLoss + RegLoss: 0.17449446 + 0.092357084 = 0.26685154\n",
      "\n",
      "\n",
      "Epoch Number: 93\n",
      "\n",
      "Classification Train Loss: 0.1467396484480964\n",
      "Training accuracy (Classification): 0.9831944546765752\n",
      "Test accuracy 0.935227\n",
      "MarginLoss + RegLoss: 0.17004617 + 0.09433146 = 0.26437762\n",
      "\n",
      "\n",
      "Epoch Number: 94\n",
      "\n",
      "Classification Train Loss: 0.1460545692178938\n",
      "Training accuracy (Classification): 0.9841666767994562\n",
      "Test accuracy 0.935227\n",
      "MarginLoss + RegLoss: 0.17442052 + 0.09421773 = 0.26863825\n",
      "\n",
      "\n",
      "Epoch Number: 95\n",
      "\n",
      "Classification Train Loss: 0.14522172489927876\n",
      "Training accuracy (Classification): 0.9843055639002058\n",
      "Test accuracy 0.936223\n",
      "MarginLoss + RegLoss: 0.16918503 + 0.09473272 = 0.26391774\n",
      "\n",
      "\n",
      "Epoch Number: 96\n",
      "\n",
      "Classification Train Loss: 0.14685245561930868\n",
      "Training accuracy (Classification): 0.9838888992865881\n",
      "Test accuracy 0.93423\n",
      "MarginLoss + RegLoss: 0.1715351 + 0.09685955 = 0.26839465\n",
      "\n",
      "\n",
      "Epoch Number: 97\n",
      "\n",
      "Classification Train Loss: 0.15079948357823822\n",
      "Training accuracy (Classification): 0.9830555634366142\n",
      "Test accuracy 0.935227\n",
      "MarginLoss + RegLoss: 0.1724481 + 0.0967999 = 0.269248\n",
      "\n",
      "\n",
      "Epoch Number: 98\n",
      "\n",
      "Classification Train Loss: 0.15230303982065785\n",
      "Training accuracy (Classification): 0.9816666767001152\n",
      "Test accuracy 0.932237\n",
      "MarginLoss + RegLoss: 0.17799449 + 0.09676037 = 0.27475485\n",
      "\n",
      "\n",
      "Epoch Number: 99\n",
      "\n",
      "Classification Train Loss: 0.1494007593848639\n",
      "Training accuracy (Classification): 0.9838888976309035\n",
      "Test accuracy 0.932735\n",
      "MarginLoss + RegLoss: 0.17286898 + 0.096531555 = 0.26940054\n",
      "\n",
      "\n",
      "Non-Zero : 4156.0 Model Size: 31.703125 KB hasSparse: True\n",
      "\n",
      "For Classification, Maximum Test accuracy at compressed model size(including early stopping): 0.94369704 at Epoch: 56\n",
      "Final Test Accuracy: 0.93273544\n",
      "The Model Directory: usps10//TFBonsaiResults/16_20_53_15_02_19\n",
      "\n"
     ]
    }
   ],
   "source": [
    "bonsaiTrainer.train(batchSize, totalEpochs, sess,\n",
    "                    Xtrain, Xtest, Ytrain, Ytest, dataDir, currDir)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
